The Entropy Message Passing: A New Algorithm Over Factor Graphs

Message passing over a factor graph can be considered as a generalization of the many well-known algorithms for an efficient marginalization of multivariate functions. The specific instance of the algorithm is obtained by selection of an appropriate commutative semiring for the range of the function to be marginalized. Some of the examples are the Viterbi algorithm, obtained on a max-product semiring and the forwardbackward algorithm obtained on a sum-product semiring. In this paper, the Entropy Message Passing algorithm (EMP) is developed. It operates over an entropy semiring, previously introduced in the automata theory. It is shown how the EMP can be used for efficient computation of the model entropy and of the complex expressions which appear in the Expectation Maximization and the gradient descent algorithms. Likewise, the EMP can be seen as a generalization of the Sum-Product Algorithm and this connection is derived.